Papers with sequential training
Adversarial Domain Adaptation Using Artificial Titles for Abstractive Title Generation (P19-1)
Copied to clipboard
| Challenge: | Obtaining good quality labeled data can be difficult and expensive for abstractive summarization models . authors propose the use of artificial titles for unlabeled target documents . |
| Approach: | They propose to use artificial titles and sequential training to capture grammatical style of unlabeled target domains to adapt to/from news articles and Stack Exchange posts. |
| Outcome: | The proposed techniques can boost performance for unsupervised adaptation and fine-tuning with limited target data. |
Mitigating the Diminishing Effect of Elastic Weight Consolidation (2022.coling-1)
Copied to clipboard
| Challenge: | Existing work addresses catastrophic forgetting in sequential training by fine-tuning pre-trained language models on different datasets. |
| Approach: | They propose to rescale the components of EWC to mitigate catastrophic forgetting by mixing new and old training data and retraining the model from scratch. |
| Outcome: | The proposed method requires smaller values for the trade-off parameters to achieve comparable results to EWC on natural language inference and fact-checking tasks. |
HFT: Half Fine-Tuning for Large Language Models (2025.acl-long)
Copied to clipboard
| Challenge: | Large language models (LLMs) with one or more fine-tuning phases can unlock various capabilities, but can be catastrophic forgetting during sequential training. |
| Approach: | They propose a method to regularly reset partial parameters to mitigate forgetting issues by using half fine-tuning instead of full fine-uning. |
| Outcome: | The proposed approach reduces the risk of catastrophic forgetting during training and the parametric knowledge lost during training may be overwhelmed by incoming training data. |
Recurrent Knowledge Identification and Fusion for Language Model Continual Learning (2025.acl-long)
Copied to clipboard
Yujie Feng, Xujia Wang, Zexin Lu, Shenghong Fu, Guangyuan Shi, Yongxin Xu, Yasha Wang, Philip S. Yu, Xu Chu, Xiao-Ming Wu
| Challenge: | Continual learning (CL) is crucial for large language models without costly retraining. |
| Approach: | They propose a framework for recurrent knowledge identification and fusion that enables dynamic estimation of parameter importance distributions to enhance knowledge transfer. |
| Outcome: | The proposed framework mitigates catastrophic forgetting and enhances knowledge transfer. |